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analyzed the results obtained and discussed the lesson learned for 
stakeholders of the telecommunications industry. Primary data was obtained 
by conducting a market survey in May 2017, with a confidence level of 95%, 


representing the 4G subscribers in the Indonesia market. There are three 





main subjective aspects of QoE: service quality, data speed, and network 
Keywords: quality (indicated by stability of signal). The analyses were conducted based 
on ACR-HR score, comparing users’ experiences of 4G with their previous 


aC : 3G subscriptions. It was found that, in general, the Indonesian market 
Indonesia perceived 4G as offering a better experience than 3G. However, one of the 
Mobile technology important lessons that can be learned is that network operators should 


QoE provide a more stable signal, as the market gave a lower score for this aspect. 


Quality of experience : ' ; : . 
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1. INTRODUCTION 

The International Telecommunication Union (ITU) has defined quality of experience (QoE) as the 
“degree of delight or annoyance of the user of an application or service” [1]. In contrast to quality of service 
(QoS) assessment, QoE assessment is mainly based on user habits (i.e. it is user centered), while QoS 
assessment is based on technical performance (technology centered) [2]. In the context of mobile 
communications networks, QoE assesses the users’ perceptions of the service, and these perceptions may 
relate to all aspects of the technical system, including terminals, networks, infrastructure, etc. Hence, QoE 
assessment is essential, since experience gained from previous services, as well as expectations for the new 
service, influence future strategic plans. For example, QoE assessment data can be used to overcome the 
problem of churn rates and the exclusion of new service plans [2], aspects of strategic capability that cannot 
be satisfied merely by utilizing QoS data [3]. 

Indonesia, the fourth most populous country in the world, commenced its commercial 
implementation of 4G mobile technology in 2015. However, public QoE assessment of 4G has not been 
officially reported since its commencement, despite the fact that QoE data is essential for supporting the 
continuing strategic plans of mobile network operators. Lessons that can be drawn from current market 
experience are no less important for the success of future technology rollouts than they are for the current 
4G service. 

This research aims to assess the quality of QoE of 4G mobile technology in the Indonesian market. 
Subsequently, lessons learned for the mobile network operators are derived. A market survey was conducted 
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in May 2017, with a minimum sample complying with Slovin’s formula, giving random sampling with a 
confidence level of 95%. The respondents were requested to give their subjective perception of 24 items 
relating to three assessment aspects, these being service quality, data speed, and stability of signal for the 4G 
network. To provide a comprehensive analysis, the ACR-HR method was used to analyze the survey data, 
represented by scores ranging from -5 (poorest) to +5 (best). Such a scoring system provides a comparison 
between users’ current experience of 4G services and their previous experience of 3G. The research theme 
presented in this paper is part of ongoing work investigating the shifting of the Indonesian market from 3G to 
4G [4], [5]. 

The remainder of the paper is structured by reviewing underlying theories in Section 2 and 
presenting the method of market survey undertaken in Section 3. The results and discussion of lessons 
learned are presented in Section 4 and the conclusion is presented in Section 5. 


2. THEORY OF QoE ASSESSMENT 

The definition of QoE given by the European Telecommunication Standards Institute is "a measure 
of user performance based on both objective and subjective psychological measures of using ICT service or 
product” [2]. In line with this, Recommendation ITU-T G.1011 sets out two overall methods to be used for 
assessing QoE: subjective assessment through perception surveys, and objective assessment through 
monitoring, planning, and/or lab testing. 


2.1. Objective Assessment 

Objective assessment of QoE involves the use of an estimation model matched to the actual 
technological service or product being reviewed. The model should comply with the service or product 
objectives, provide the new application features, and have the new network configurations. In regard to 
communications technology, objective QoE assessment can be carried out in three ways: intrusive mode, 
non-intrusive mode, and planning mode [6]. In intrusive mode, a signal is injected into the system under test, 
with the aim of measuring the output signal. Non-intrusive mode assessments measure traffic directly without 
adding a signal to the system, while assessment in planning mode is not performed in a real system and 
usually serves as a tool for designing future network systems. Such a mode requires a planning-specific 
assumption as well as detailed knowledge about the service or product being assessed [6]. Objective 
assessment is cheaper and faster than subjective assessment, but the accuracy of the results obtained depends 
on the suitability of the model. Hence, objective assessment is considered to perform with limited validity 
and reliability. Similar to QoS assessment [7], such an objective assessment would focus on the QoE 
technical metrics of a service application. The example of objective QoE assessment would be a laboratory- 
based assessment of a new video service before it is commercially launched to the market [8] and the PSNR 
evaluation of video application in wireless multimedia sensor network [9]. In relation to the 4G network, the 
Office of Communication (OFCOM) in the UK conducted a data speed test and reported that the UK 4G 
network performed at a faster data speed than the 3G network [10]. 


2.2. Subjective Assessment 

This research is developed based on subjective assessment. Subjective assessment of QoE measures 
experience, perception, and satisfaction as they are defined by users. Users are asked to provide opinions 
about the quality of various aspects of a service or technology, expressed using a rating scale known as a 
mean opinion score, or MOS. ITU defines MOS as “the value on a predefined scale that a subject assigns to 
his opinion of the performance of the telephone transmission system used either for conversation or for 
listening to spoken material” [11]. Users respond to questions about their levels of satisfaction with service 
operations through survey techniques such as interviews, discussion groups, and questionnaires. In 
comparison with objective methods, subjective QoE assessment requires higher budgets and more extensive 
resources. Subjective assessment of QoE can also measure overall system performance by taking into 
account technical parameters (QoS). Hence, QoE data can combine knowledge of the users’ subjective 
experiences with the system’s technical parameters. An example of this method is the QoE measurement of 
the 3G network in the UK [12] conducted by OFCOM, one important finding of which was that users were 
most concerned about the availability of voice-call services and clarity of sound. Another example of the 
QoE subjective assessment includes the evaluation of users’ perception of computer games application [13]. 
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3. METHOD 

3.1. Primary Data of QoE Subjective Assessment 

A survey is a much-used tool for obtaining primary data for subjective QoE assessment. For this 
study, a market survey was conducted in May 2017, questioning respondents using a random face-to-face 
method. The number of respondents in the sample complies within Slovin’s formula to represent a greater 
than 1 million population with a margin of error of 5%. The main requirements for the chosen respondents 
were as follows: 

a. Already be subscribed to 4G mobile service (either as prepaid or postpaid users) for a minimum of one 
day. 

b. Be resident in Jakarta (the capital city of Indonesia), with his/her main routine activities being located in 
the city. 

c. Have subscribed to a 3G service before using 4G technology. 

d. Use a mobile handset as the terminal to access 4G service (in some countries, including Indonesia, 4G is 
also used for fixed-data communication services). 

The sample area was Jakarta, a city with a population of around 12 million. Previous research 
carried out by the authors has argued that Jakarta can be used as a sampling area representing the national 
market because [4]: 

a. Jakarta is the capital city, the ‘life style’ center of the country, the most multicultural city in Indonesia, 
and the center for modern trends that are subsequently followed by other parts of the country. 

b. Typically, the national operators launch their latest services, including 4G, in Jakarta, and penetrate into 
other service areas afterwards. 

c. Despite the heterogeneity of the country’s demographic profile, around 66% of the population of Jakarta 
is Javanese. This proportion is similar to the overall distribution of population in Indonesia. 

d. Some 60% of the Indonesian money supply is located in Jakarta, 30% in other cities and 10% in rural 
areas. Therefore, all mobile network operators have treated mobile subscribers in Jakarta as their main 
revenue sources, taking into account both their dynamic and routine activities (the term ‘money supply’ 
refers to the amount of money available in an economy). 


3.2. Questionnaire Design 

Factors that influence QoE are any characteristic of a user, system, service, application, or context 
whose actual state or setting may have an influence on QoE for the user [3]. In this research, mobile users 
were requested to respond to 24 statements grouped into three assessment aspects: service quality, data 
speed, and stability of signal. These assessment aspects represent factors influencing QoE. For each 
statement, respondents were asked to give a score representing their subjective evaluations of 4G and 3G 
mobile subscriptions. The scale used was an absolute category rating (ACR) of 5 points, graded as follows: 
5=very good, 4=good, 3=normal/ordinary, 2=poor, and l=very poor. Table 1 gives a selection of the 
statements included in the questionnaire. 


Table 1. Example Questionnaire Statements 





STATEMENT pCESCINACES 4G service 


SUE SER (currently used) 





SERVICE QUALITY ASPECTS 
The sound quality when making traditional phone calls 
(TDM-based) 


2 Quality of sending and receiving SMS and MMS 


The quality of voice communication/voice using 
3 messenger applications (such as WhatsApp, LINE, 
etc.) 


DATA SPEED ASPECT 


Data speeds while downloading / download files from 
emails, websites 1 2 3 4 5 1 2 3 4 in) 


Data speeds when uploading / upload files from 


10 emails, websites 1 2 3 4 5 1 2 3 4 5 


Delivery speed of text using messenger applications 
(such as WhatsApp, LINE, etc.) 1 2 3 4 5 1 2 3 4 5 


STABILITY OF SIGNAL 


11 
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Table 1. Example Questionnaire Statements 





STATEMENT AGISEnVICES 4G service 


eraser (currently used) 





Signal stability while downloading / download files 


oa from emails, websites 1 2 3 4 5 1 2. 3 4 5 
%B Signal stability while uploading / upload files from 

emails, websites 1 2 3 4 5 1 2 3 4 5 
A Signal stability while Wi-Fi tethering (handset as a 


modem / personal hotspot) 1 2 3 4 55 1 2 3 4 55 





In the questionnaire, respondents are requested to express what they perceive individually for both 
3G and 4G service. Using the respondents’ scores, absolute category rating with hidden reference (ACR-HR) 
was calculated by subtracting the 3G scores from the 4G scores. The difference between the MOS value of 
the 4G service and the 3G service is an adaptation of the Recommendation ITU-T P.910. ITU defines the 
ACR-HR method as a category judgment where test sequences are presented one at a time and are rated 
independently on a category scale [14]. Since the main QoE assessment being sought is for 4G the reference 
contained in the procedure is 3G service. 

The ACR-HR scale used in this research to translate the qualitative comparison values is as follows: 
ACR-HR range from -5 to -2: 4G service is perceived as much worse than 3G 
ACR-HR range from -2 to -1: 4G service is perceived as worse than 3G 
ACR-HR equal to 0: 4G service is perceived as just the same as 3G 
ACR-HR range from | to 2: 4G service is perceived as better than 3G 
ACR-HR range from 2 to 5: 4G service is perceived as much better than 3G 


ep aooP 


4. RESULTS AND DISCUSSION 
4.1. Basic Respondents’ Data 

Table 2 presents the basic data for the respondents. It shows that the majority of respondents were 
male. The age of respondents closely follows a normal distribution, and sampling is predominantly of those 
aged 19 and 20. The data obtained shows that 86% of respondents had used 4G for more than three months 
while 71.09% of respondents used the same mobile phone brands for both 3G and 4G services. 


Table 2. Basic Information for Respondents 

No Data Percentage 
51.42 % male 

: rides 48.58 % female 
Age range from 9 to 50, predominantly 19 and 
20 
86% more than 3 months 
14 % less than 3 months 
71.09% using the same brand for both 4G and 
3G 
49.76 % bachelor degree 








2 Age 
3 Length of time as a 4G subscriber 
+ Brand of 4G handset 


Educational level (including ongoing 


5 18.25 % diploma/vocational degree 
study or already graduated) 27.01 % high school 
61.37% student 
6 Oeeupaion 23.70% private employee 


4.7 % state employee 
0.47 % other 

36.26 % Telkomsel 

31 % Indosat 

25.83 % XL 

4.7% Three 

1.6 % Smartfren 

0.47 % Axis 

38.5 % at home 
Locations most frequently used to 27.2 % at school or office 
access 4G services 22 % in public facilities 
12.1 % in vehicles (while traveling ) 


7 4G operators subscribed to 
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The survey found that students and private company employees (including state-owned companies 
and foreign employees) constituted the majority of 4G service respondents by profession. This suggests that 
students are the respondents who most easily accept new technology. In terms of network operators, the 
results mirrored the actual market of six main operators, in which Telkomsel is a dominant national operator 
with a market share of more than one third. 

The basic data also shows the locations most frequently used to access 4G services. The most 
commonly used location for accessing 4G was at home. Mobile phones are telecommunication technology 
innovations developed for mobile use but are currently used to access information and telecommunications 
most often when mobile phone users are at home. 


4.2. Results of the Questionnaire 
QokE for Service Quality Aspects 

Figure | lists details of service quality aspects with their ACR-HR score results. Overall, it can be 
seen that none of the scores exceed 1.5. The results signify that 4G service qualities are perceived as “better” 
than 3G, but with the lowest ACR-HR scores given for traditional voice-call sound quality and SMS service 
quality. These results are not surprising since voice calls in the Indonesian mobile network are still carried on 
the 2G network, 4G-based voice technology (such as VoLTE) having not yet been implemented. The current 
voice-call platform is circuit-switched (CS) fallback, which enables the provision of voice and traditional 
circuit-switched services. Figure | also indicates that users appreciate the significant service improvements 
provided by 4G for video calls and multimedia streaming (such as YouTube, JOOX, Spotify, etc.) suggesting 
that significant technical advance has provided a better experience for 4G subscribers. Additionally, the 
Indonesian market also appreciates the improvement of the sound quality for voice calls using messenger 
applications. However, the average score for this aspect is quite low (1.0 out of 5.0), indicating a small level 
of improved QoE when compared with 3G. 


The service quality of other applications 


The service quality when accessing game online 


The service quality when watching CCTV / monitoring online 
The service quality when using multimedia streaming 
The service quality of video call 


The sound quality when making voice call using messenger-apps (e.g. 
WhatsApp, LINE, etc.) 


The service quality when sending/receiving SMS and MMS 





The sound quality when making traditional phone calls 


0.0 os 10 15 


ACR-HR score 


Figure 1. The QoE results for service quality aspects 

4.2.2. QoE for Data Speed Aspects 

Figure 2 provides details of the data speed aspects and their ACR-HR score results. Overall, it can 
be seen that the scores range from 1.0 to 1.5. All the 4G data speed aspects are perceived as “better” than 3G 
data speed, while no single experience is perceived as “worse”. The maximum ACR-HR scores are given for 
the experience of uploading files, downloading files and Wi-Fi tethering (using handsets as personal 
hotspots). The results indicate that current implementation has fulfilled the basic purpose of 4G technology, 
which is to enhance data communications over previous mobile technology evolutions. Technically, 4G 
should accomplish data communications of up to 1 Gbps in stationary states and 100 Mbps in moving 
conditions. 
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Data speed when activating Wi-Fi tethering (handset as 
personal hotspot) 


Data speed when accessing other applications 

Data speed when accessing online taxi application 

Data speed when accessing navigation applications 

Data speed when watching multimedia streaming 

Data speed when accessing social media 

Data speed when accessing Internet webpage (browsing) 
Data speed when sending/receiving files in messenger-apps 
Data speed when sending/receiving text in messenger-apps 
Data speed when uploading any files 


Data speed when downloading any files 





0.0 05 1.0 15 2.0 
ACR-HR score 


Figure 2. The QoE results for data speed aspects 


However, we cannot judge the current subjective QoE as inferior, because the results indicate a 
score of “better”, although the score for “better” is only just beyond the “same as before” range. In fact, most 
of the Indonesian market has enjoyed 3.5G technology (such as high-speed packet access, or HSPA), which 
can, in some cases, give data speed of hundreds of Mbps. 


QoE for Network Quality Aspects 

Figure 3 provides details of the network quality aspects and their ACR-HR score results. Overall, it 
can be seen that the scores do not exceed 1.2. The subjective indication of network quality is represented by 
the term “stability of signal”. Mobile subscribers can easily judge this based on technical indicators in their 
handsets. The results show that the stability of 4G signals is perceived as “better” than 3G services. However, 
the network quality for traditional voice calls has a score of less than 1.0. Such a near-to-zero score indicates 
that users’ experience of 4G is almost the same as their experience of 3G. This is consistent with the QoE 
results for the service quality aspect, since the voice-call network is still running on the 2G network (using 
the underlying CS fallback mechanism). Another interesting finding is that the Indonesian market regards 
data service as being more stable than voice service. This is, therefore, a reliable indicator for mobile network 
operators to consider the implementation of voice communications services, such as VoLTE, which are fully 
delivered over the 4G data network. 


Stability of signal when activating Wi-Fi tethering (handset 
as personal hotspot) 


Stability of signal when uploading any files 


Stability of signal when downloading any files 


Stability of signal when making voice call using messenger- 
apps (e.g. WhatsApp, LINE, etc.) 


Stability of signal when making traditional phone call 





o6 os 


ACR-HR score 
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Figure 3. QoE results for network quality aspects 


5. LESSONS LEARNED 
In the final part of the questionnaire, the respondents were asked to give general QoE conclusions 
for their current 4G subscription. The results indicated that 59.48% of respondents perceived 4G service as 

“better” than 3G, while 34.36% of respondents perceived 4G service is “much better” than their previous 3G 

service. Hence, Indonesian operators can be seen to have delivered significant performance improvements for 

4G services, in that users notice positive differences and perceive them as being better than 3G. For each 
assessment point investigated, the data speed and network quality aspects provided positive ACR-HR scores. 

Such data should be easily understood and translated into strategic references for mobile network operators. 

QoE data is also useful for optimizing products and/or services [2], because technical teams may not have 

had sufficient knowledge about the specific technical parameters that affect user experience. Some of the 

main points that can be further elaborated as strategic references are as follows: 

a. Operators should pay special attention to the aspect of network quality, since stability of signal receives 
a lower ACR-HR score. This survey result may also indicate that subscribers have experienced some 
blank spots as well as unstable coverage. One possible cause would be that, as the development of 4G 
started only a few years ago, 4G coverage is not as extensive as 3G coverage. It is therefore suggested 
that Indonesian operators should accelerate infrastructure rollout. 

b. QoE data is also useful for optimizing and improving current data services. Technical teams often focus 
on the product, but do not have much knowledge about the specific technical parameters that affect end- 
user experience. It is perhaps believed that user experience is a subjective measure, but it can also be 
analyzed objectively. In other words, QoE, which reflects the measurement of user's psychological 
feelings, should also be expressed in terms of QoS characteristics. In this way, data for QoE can be 
combined with technical QoS parameters, providing valuable information about user experience. For 
example, this QoE assessment has indicated that most subscribers are actively using messenger 
applications such as WhatsApp for voice-call communications. Any poorer technical performance, such 
as longer delays and lower power signals, will cause a degraded sound quality that is subjectively 
perceived as inconvenient by users. Hence, technical quality enhancements are needed to satisfy users’ 
subjective experiences if voice communications services are to be preferred over messenger applications. 

c. QoE data can help in overcoming the problem of high churn rates. Profits and brand image will be 
negatively affected if service consistently fails to meet consumer expectations. Opportunities to develop 
customers’ loyalty will be increased by developing improvements in current services with the help of 
QoE data. For example, the survey shows that customers value data speed for mobile tethering (using 
mobile handset as Wi-Fi hotspots). Hence, operators should pay attention to such aspects of market 
behavior in Indonesia, where consistent development of the relevant service will be likely to increase 
brand loyalty. On the other hand, a chain of negative reactions may easily occur when the mobile 
network operators, as well as service developers, are unaware of such phenomena. 

d. QoE can be used as a reference for future service applications. Future services will be adopted by the 
market, or on the other hand, rejected by the market. One reason for rejection is that consumers can 
experience a lower service experience when they start to utilize a new service. Such conditions may 
occur even if the service has received positive feedback during market testing and/or pre- 
commercialization advertising. Such a rejection can be predicted and avoided if QoE data is analyzed 
accurately and in detail through user testing before the product is launched. 

e. QoE assessment has also shown that subscribers appreciate faster data, suggesting that operators should 
prioritizing relevant data communication applications. However, mobile network operators need to 
survive in the era of OTT (over-the-top) content. This presents a challenge, as mobile operators have had 
to invest to build infrastructure, but do not enjoy significant revenue from the commercialization of 
applications, because subscribers are paying for the data network, not the service applications. 


6. CONCLUSION 

This paper has presented QoE assessment results for the 4G mobile technology that has been rolled 
out in the Indonesian market. A market survey has been conducted with a confidence level of 95%, 
investigating the 4G subscriber in the Indonesia market. Three main subjective aspects of QoE assessment 
have been investigated: service quality, data speed and network quality (indicated by stability of signal). The 
analyses were conducted based on ACR-HR score, comparing users’ experiences between 4G and their 
previous 3G subscriptions. For the service quality aspect, the ACR-HR scores were positive but never 
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exceeded 1.5. This result signifies that 4G service qualities are perceived as “better” than 3G, but with the 
lowest ratings in this aspect for traditional voice calls and SMS. For the data speed aspect, the scores ranged 
from 1.0 to 1.5, indicating the services are perceived as “better” than 3G data speed. In this aspect, the 
maximum ACR-HR scores were given for the experience of uploading files, downloading files and Wi-Fi 
tethering (using handsets as personal hotspots). For the network quality aspect, the QoE assessment had no 
scores that exceed 1.2, with the network quality of traditional voice calls having a score of below 1.0. Such a 
near-to-zero score indicates that the current experience for making traditional voice calls is felt to be no 
different to the previous 3G service. To conclude, based on QoE assessment results for these three aspects of 
user experience, the Indonesian market perceives 4G as providing a better experience than 3G service. The 
QoE assessment results can be further developed into strategic lessons for mobile network operators. From 
these five lessons, a particularly important issue highlighted is that Indonesian operators should pay special 
attention to the aspect of network quality, since the ACR-HR score for stability of signal is low. 
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